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ABSTRACT 


The Elo rating system has been recognised as an effective 
method for modelling students and items within adaptive ed- 
ucational systems. The existing Elo-based models have the 
limiting assumption that items are only tagged with a single 
concept and are mainly studied in the context of adaptive 
testing systems. In this paper, we introduce a multivari- 
ate Elo-based learner model that is suitable for the domains 
where learning items can be tagged with multiple concepts, 
and investigate its fit in the context of adaptive learning. To 
evaluate the model, we first compare the predictive perfor- 
mance of the proposed model against the standard Elo-based 
model using synthetic and public data sets. Our results from 
this study indicate that our proposed model has superior 
predictive performance compared to the standard Elo-based 
model, but the difference is rather small. We then inves- 
tigate the fit of the proposed multivariate Elo-based model 
by integrating it into an adaptive learning system which in- 
corporates the principles of open learner models (OLMs). 
The results from this study suggest that the availability of 
additional parameters derived from multivariate Elo-based 
models have two further advantages: guiding adaptive be- 
haviour for the system and providing additional insight for 
students and instructors. 


1. INTRODUCTION 


Adaptive educational systems make use of data about stu- 
dents, learning process, and learning products to adapt the 
level or type of instruction for each student. Commonly, 
this adaptivity takes the form of selecting items from a 
large repository of learning resources to match the current 
learning ability of a student [17]. To do so, adaptive ed- 
ucational systems rely on learner models that capture an 
abstract representation of a student’s ability level based on 
their performance and interactions with the system [6]. Two 
conventional approaches have been heavily studied for mod- 
elling learners. (1) Bayesian Knowledge Tracing (BKT) uses 
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a Hidden Markov Model for capturing students’ knowledge 
state as a set of binary variables indicating whether a knowl- 
edge component has been mastered [4]. (2) Item Response 
Theory (IRT) [11] and its extensions such as Additive Factor 
Model (AFM) [3] and Performance Factor Analysis (PFA) 
[18] rely on a logistic regression model to estimate latent 
traits related to students’ knowledge state and the difficulty 
of learning items. Neither of these approaches, however, can 
be easily integrated into online adaptive educational systems 
as they generally require pre-calibration on big samples of 
data and ongoing addition of new students and new learn- 
ing items to the system necessitates continuous calibration 
of model parameters [19]. 


The Elo rating system has been shown to be an effective 
alternative to the above mentioned conventional approaches 
for modelling students in adaptive educational systems [14]. 
It is simple, fast, robust and order-sensitive which makes it a 
suitable model for adaptive educational systems where it is 
required to update students’ proficiency level upon admin- 
istration of each question [23]. In the educational context, 
the Elo-based model is employed to conduct a paired com- 
parison among students and learning items as competitors 
[23]. This model is self-correcting, meaning that the ratings, 
in the long run, should correctly reflect students’ knowledge 
states and difficulty levels of questions [20]. 


The majority of the existing studies on Elo-based models 
have the following two characteristics: They (1) use repos- 
itories that contain items that are pure [9] and are tagged 
with a single concept and (2) are studied in the context 
of adaptive testing systems such as computerised adaptive 
testing (CAT) [5]. The contribution of this paper lies in (1) 
introducing a new variant of the Elo-based algorithm called 
M-Elo that has the capacity to model students and items us- 
ing repositories that contain items that are tagged with one 
or more concepts, and (2) investigating its fit in the context 
of adaptive learning systems, which in contrast to adaptive 
testing systems, take a more student-centred approach with 
the aim of assisting students in their learning. 


To evaluate the applicability of M-Elo in the context of adap- 
tive learning where learning items are explicitly tagged with 
one or more concepts, we first compare the predictive per- 
formance of M-Elo against the standard Elo-based model us- 
ing simulated data sets. The results from these experiments 
demonstrate that the behaviour of the M-Elo is consistent 
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with expectations over a range of parameter settings, and 
therefore this provides some evidence to suggest that it will 
be robust in a real-world setting. We then compare the pre- 
dictive performance of M-Elo against the standard Elo-based 
model using public data sets. The results from the experi- 
ments indicate that M-Elo have superior predictive perfor- 
mance compared to the standard Elo-based model, but the 
difference is rather small. Finally, we integrate M-Elo into 
an adaptive learning system and conduct a case study by 
piloting this system in a large introductory course on rela- 
tional databases at The University of Queensland. 


The results obtained from our case study suggests that using 
M-Elo, as a multivariate Elo-based model, in adaptive learn- 
ing systems have two additional advantages beyond the stan- 
dard Elo-based model: The first advantage lies in the avail- 
ability of additional parameters that provide insight into the 
characteristics of the domain and the learning process which 
in turn can be used for guiding adaptivity. The second ad- 
vantage is that when M-Elo is opened based on the concepts 
of open learner models (OLMs) [2], it can provide additional 
insight on course level and individual level competencies and 
gaps, that can be used by instructors for improving learning 
item design, while also providing meta-cognitive benefits for 
students, such as increased motivation and trust in the sys- 
tem. The conducted case study also revealed some of the 
shortcomings of using Elo-based models in the context of 
adaptive learning. The main issue is that Elo-based models 
consider items and students as identical rivals. This assump- 
tion seems to fail to adequately model the long-term use of 
an adaptive learning system in which students’ knowledge 
may increase over time while the difficulty level of the items 
remains constant. 


2. BACKGROUND 


Elo-Based Learner Models. The Elo rating system is orig- 
inally developed to rate chess players and is established 
based on paired comparison of data where two chess players 
compete against each other [23]. In the educational set- 
ting, a similar paired comparison can be conducted between 
a student and a question being attempted by the student. 
The standard implementation of the Elo-based model in ed- 
ucation resembles the Rasch model employed in IRT that 
models students and questions with a single global parame- 
ter [20]. An important extension to the Elo-based model in 
education is the multivariate Elo-based model proposed by 
[9] in the context of psychometrics, where instead of using 
a global knowledge parameter for students, it uses an over- 
lay model which estimates the competency of a student in 
each different concept using a separate parameter. A similar 
model for adaptive practice of facts was later proposed by 
[20], which had an additional global parameter compared to 
the multivariate Elo-based model that was used in combi- 
nation with the concept level parameters in modelling the 
ability of a student on each concept. Both of these models 
make the assumption that items are tagged with a single 
concept, which limits their applicability in domains where 
items can be tagged with multiple concepts. For example, 
in a Programming domain, an arbitrary learning item might 
be associated with both ”lists”and ”loops” concepts. M-Elo, 
our proposed model, is an extension over the multivariate 
Elo-based model that has the capacity to model students 
and items in the presence of items with multiple concepts. 


Adaptive Testing vs. Adaptive Learning. Adaptivity in 
educational systems has been investigated broadly both in 
the context of computerized adaptive testing (CAT) and 
adaptive learning [17]. Generally, adaptive testing systems 
conduct an exam using a sequence of questions that are suc- 
cessively administered with the purpose of maximising the 
precision of the system’s current estimate of the student’s 
ability. The exam is usually terminated once the system has 
an estimate of the student’s ability with a confidence level 
that exceeds a user-specified threshold. In contrast, adap- 
tive learning systems such as ALEKS! take a more student- 
centred approach with the aim of assisting students in their 
learning. As such, in most adaptive learning systems, stu- 
dents (1) have the opportunity to decide whether or not to 
engage with the suggested learning items, (2) can spend, 
theoretically, an infinite amount of time on a learning item 
or on the system and (3) will receive rich feedback on their 
learning after engaging with each learning item. In this pa- 
per, we examine the fit of our proposed model, and more 
generally multivariate Elo-based models in the context of 
adaptive learning. 


Open Learner Models. Open learner models (OLMs) are 
learner models that are externalised and made accessible 
to students or other stakeholders such as instructors, often 
through visualisation, as an important means of supporting 
learning [2]. They have been integrated into a variety of 
learning technologies such as learning dashboards [1], intel- 
ligent tutoring systems [21] and adaptive learning platforms 
[8] to help students and instructors in monitoring, reflecting 
and regulating learning [2]. In this paper, we investigate 
the benefits of opening Elo-based models, such as M-Elo 
in adaptive learning system based on the principles of the 
OLMs. 


3. ELO-BASED LEARNER MODELLING 


In this section, we first define a mathematical notation for 
describing the models. Section 3.1 then provides a review 
of the standard Elo-based model in the educational context. 
Finally, Section 3.2 introduces our proposed variation of the 
multivariate Elo-based model, which we call M-Elo. 


In what follows, let Un = {ui...un} denote a set of stu- 
dents who are enrolled in a course on an adaptive educa- 
tional system, where u,, refers to an arbitrary student. Each 
course consists of a set of concepts Ar = {61 ...61}, referred 
to as the domain model, where 6; presents an arbitrary con- 
cept. In this work, the notion of a concept is based on 
taxonomies of knowledge components described by [16]. Let 
Qu = {m.--qu} present the content model, denoting a 
repository of learning items that are available to students 
in a course in the adaptive learning system, where qm refers 
to an arbitrary item. These learning items can be tagged 
with one or more knowledge components; Qayrx1 is a two- 
dimensional array, where w,,; is 1/g if item qm is tagged with 
g knowledge components including knowledge component 61, 
and 0 otherwise. Let a two-dimensional array Any keep 
track of students’ attempts on the items, where dnm = 1 
indicates that student u,, has answered item q,, correctly, 
and Gnm = 0 indicates that student un has answered item 
Gm incorrectly. 


https: //www.aleks.com/ 
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3.1 The Standard Elo-based Learner Model 


(Elo) 

In the standard Elo-based learner model (Elo), both stu- 
dents and items are considered as identical rivals. Elo as- 
sumes one student parameter 0, indicating u,,’s global profi- 
ciency level on the entire domain and one item parameter dm 
presenting the difficulty level of item gm. It uses a logistic 
function to estimate the probability of a correct answer by a 
student to a given item based on the difference between the 
student’s global proficiency level and the item’s difficulty. 
[20]: 


P(dnm = 1|0n,dm) = 0(0n — dm) (1) 


To account for the guessing effect in the case of multiple 
choice items with c possible options, Formula 1 can be easily 
replaced with a shifted logistic regression using: P(anm = 
1]0n,dm) = ++(1—+4)*0(0,—dm). once Un has attempted 
dm, the knowledge state of un and difficulty level of gm are 
simultaneously updated using the following formulas: 


On = On + K(anm — P(anm = 1|0n,dm)) (2) 


din = dm + K(P(anm = 16n,dm) — nm) (3) 


, where K is a constant value determining the sensitivity of 
the estimations based on the student’s last attempt. The 
updates to the student’s knowledge state (Formula 2) and 
the difficulty of the item (Formula 3) in Elo follows the prin- 
ciples of zero-sum game, in which the sum of gains (loss) to 
the student’s knowledge state and the loss (gain) to the dif- 
ficulty of the item after the student answers the item turns 
out to be zero. In most extensions of the Elo-based mod- 
els, in order to get to a stable estimations for the student’s 
knowledge state and item difficulty, kK is replaced with an 
uncertainty function 


~y 
Cys. —— 4 
=a (4) 
, where y and @ are constant hyper-parameters determining 
the starting value and slope of changes, respectively, and n 
indicates the number of prior updates on student’s knowl- 
edge state or item difficulty [20]. 


3.2 Multi-Concept Multivariate Elo-based 
Learner model (M-Elo) 


In contrast to Elo, where only one parameter is used to 
model a student’s knowledge state on the entire domain, 
M-Elo uses independent parameters to model the student’s 
knowledge state on each individual concept in the domain, 
and a global parameter for modelling each item. As in 
Elo, for each learning item qm there is a global difficulty 
dm approximating the difficulty level of the item. For stu- 
dents, let a two-dimensional array Anx ww represents a stu- 
dent’s Elo-based learner model, where Ani represents student 
un’s knowledge state on concept 6;, approximating the pro- 
ficiency level of the student on that certain concept. To es- 
timate the probability that student un answers an item gm 
correctly, we first compute Mins era Ant X Wmi, which 
estimates u,,’s average competency on concepts that are as- 
sociated with gm. We then compute the probability of un 
answering gm correctly using: 


P(@nm = 1|nm; dm) = o(Anm — dm) (5) 


After un answers gm, the updated estimate of d», is obtained 
using: 


dm ?= dm + K(P(aam = 1|Anm, dm) — Onm) (8) 


where K can be replaced with an uncertainty function U(n) 
presented in Formula 4. To update student u,,’s Elo ratings 
based on the given answer to item qm, we update the stu- 
dent’s parameter on each concept 6; the question is tagged 
with separately using the following formula: 


Ant = Ant + aK (dam = P(anm = 1Ant, dm)) (7) 


where a is a normalisation factor, ensuring that the zero-sum 
game principles are enforced in the model. As such, the net 
change made to the parameters estimating u,,’s proficiency 
level in the concepts that are associated with gm (computed 
by Formula 7) and d» (computed by Formula 6) sum to 
zero. @ is computed using the following formula: 


|P(anm az 1[Anm; dm) = Gin 
Dies (lanm — P(dnm = Ant, dm) X wmil) 


4. EVALUATION 


In this section, we evaluate the suitability of M-Elo in the 
context of adaptive learning systems. Section 4.1 compares 
the predictive performance of M-Elo against Elo using a 
suite of simulated data sets. Section 4.2 compares the pre- 
dictive performance of M-Elo against Elo using publicly avail- 
able data sets. As commonly used in the evaluation of 
learner models, we use the area under the curve (AUC), 
root mean squared error (RMSE) and accuracy (ACC) for 
reporting the predictive performance of the models. For all 
experiments, students’ knowledge states and item difficul- 
ties in Elo and M-Elo are initialised to zero. Section 4.3 
then reports the results of a case study that integrates M- 
Elo into an adaptive learning system which incorporates the 
principles of OLMs. 


(8) 


4.1 Synthetic Data Sets 


Synthetic data sets were used to assess the behaviour of 
the models under different settings by varying parameters 
in the data generation template. The synthetic data sets 
were generated using a sequence of steps proposed by [13]. 
At first, a set of students with predefined knowledge states 
over a set of knowledge components were created. Assigning 
students’ knowledge state was performed by sampling from 
a normal distribution, where the mean of distribution for 
each student was sampled from a uniform distribution. In 
this model, the standard deviation (oc) of the normal distri- 
bution determined the complexity of a student’s knowledge 
state, where smaller values of o led to having students that 
had roughly the same ability across all of the knowledge 
components and bigger values of o led to having students 
with a higher diversity on their abilities across all of the 
knowledge components. Then, a set of learning items with 
pre-defined concepts, level of difficulty and discrimination 
was generated. Assigning concepts to items was performed 
by sampling from a discrete uniform distribution, while dif- 
ficulty and discrimination were sampled from a normal dis- 
tribution. Lastly, to compute the probability that a student 
Un answered a learning item qm correctly, a 2PL Item Re- 
sponse Theory (IRT) [10] model was used as recommended 
by [7] using: ml ——,, where, @n represents a stu- 


dent’s average competency on the concepts associated with 
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the item, and b,, and am were the difficulty level and the 
discrimination level of the item qm, respectively. In all syn- 
thetic data sets, 100 students, 1000 learning item and 70,000 
answers were sampled completely at random. Each learning 
item was tagged with one to three concepts. In these experi- 
ments, 70% of data was used for training and the remaining 
30% was reserved for the test. Each experiment was re- 
peated five times and the reported values are the average 
results across the five runs. 


— b= 10 L = 100 
0.84} 0.84 
0.82} 0.82 
re) 
2 | 
< 0.80] 0.80 
0.78 0.78 
0.0 0.2 04 0.6 0.8 0.0 0.2 0.4 0.6 0.8 
o Go 


Figure 1: AUC as o is increased. Results are re- 
ported for L = 10, and L = 100 


Figure 1 compares the AUC performance of Elo and M- 
Elo in estimating students’ knowledge states with respect 
to different values of o under two different settings for the 
number of knowledge components (LZ). Our results for L = 
10 suggest that for smaller values of 0, Elo outperforms M- 
Elo. This was predictable, as in this setting, each student 
has roughly the same competency level on all of the concepts, 
and since Elo relies on a global knowledge estimation for 
students proportional to their overall performance, it can 
outperform M-Elo in this setting. As o is increased and 
students with more diversity in their abilities across different 
concepts are generated, M-Elo outperforms Elo as it is able 
to discriminate between students’ abilities on each individual 
concept, leading to more accurate estimations of students’ 
knowledge state compared to Elo. By increasing L to 100, 
the same trend is observable; however, the intersection point 
for 0 where M-Elo outperforms Elo becomes bigger, as in 
this scenario, with the same amount of data, M-Elo needs 
to learn many more parameters independently. Evaluations 
using RMSE and ACC led to very similar patterns as AUC. 
Therefore, in the interest of space, figures reporting these 
results have not been included. 


4.2 Public Data Sets 
Three data sets namely ’Algebra I 2005-2006’ (Alg2005), ’Al- 
gebra I 2006-2007’ (Alg2006) and ’Bridge to Algebra 2006- 


2007’ (BAlg2006), which were obtained from the PSLC Datashop 


were used [15]. These data sets were originally obtained 
from Carnegie Learning’s Cognitive Tutor and were made 
available as the "DevElopment” sets in KDD Cup 2010 [22]. 
Cognitive Tutor provides a fine representation of knowledge 
components associated with each item. It is a formative as- 
sessment tool, where each step taken by a student to answer 
a problem is considered as an individual interaction. Each 
learning item (referred to as Step’ and presented by ’Step 
Name’ in the data sets) is associated with one or more con- 
cept (KC) covered in the course. We used the train/test 
split provided by KDD Cup 2010 and discarded interactions 
with items that have not clearly been tagged with particu- 
lar concepts. No students were discarded. Overall informa- 


tion about these data sets are presented in Table 1. A grid 
search was conducted to determine optimal values for hyper- 
parameters y and § of the uncertainty function, described 
in Formula 4. As also reported by [20], the results were not 
really sensitive to changes from these parameters. In all the 
reported experiments on public data sets, y was set to 1.8 
and £8 was set to 0.05. 


Table 1: Public data sets 


Data Set Students | KC Items multi-KC? 
Alg2005 575 112 | 147,914 51,171 
Alg2006 1840 714 | 319,151 21,415 


Interactions 
609,979 
1,825,030 
1,822,697 


BA1g2006 1146 493 19,954 1,650 


Table 2 compares the AUC, RMSE and accuracy (ACC) of 
the model fit statistics related to each model for estimating 
students’ knowledge state. As it is indicated, on all three 
data sets M-Elo outperformed Elo in predicting student per- 
formance, but the difference was rather small. Considering 
the insights obtained from the experiments with synthetic 
data sets, where M-Elo was outperforming Elo with a small 
margin, it may be possible to hypothesise that students of- 
ten have different competency levels on different concepts, 
but these differences are often not too significant. 


Table 2: AUC, RMSE and ACC for public data sets 


DataSet [AUC RMSE TACO 


= Elon ANE Blo— [SE lo-4/5. Me Eos [Bless [se Plos| 
ATg2005 0.750 || 0.392 | 0.385 || 0.787 


[Alge2006_[0.687_[ 0.695 [0.394 |0.300_][ 0.784 [0.797 
[ BATE2008 [0.676 [0.712 |] 0.368 | _0.361_|[ 0.827 | 0.825 


4.3 Case Study 


To investigate the fit of M-Elo for adaptive learning in an 
authentic environment, we integrate M-Elo into an adaptive 
learning system called RiPPLE [12] and piloted the plat- 
form in an introductory course on relational databases at 
The University of Queensland. The course covers many con- 
cepts that are generally included in an introductory course 
on relational databases including conceptual database de- 
sign using ER diagrams, relational models, functional depen- 
dencies, normalisation, relational algebra, Structured Query 
Language (SQL), data warehousing and database security. 
The platform was used for 13 weeks; during this period, 521 
of the students enrolled in this course made 91,340 attempts 
on 1,632 learning items which were available in the system. 
Among these items, 144 items were tagged with two or more 
of the 17 concepts, which were associated with the course. 
Our aim was to investigate the benefits and shortcomings of 
using M-Elo in an adaptive system where the model is shared 
with the students based on the principles of OLMs through 
a visualisation widget, as indicated in figure 2, which allows 
students to visually see their current knowledge state on all 
concepts of the course. This visualisation is updated and 
represented to students as soon as they answer a new item 
from the item pool. 


Guiding adaptivity. To guide the adaptivity of the system, 
the estimated knowledge states of the students and difficul- 
ties of the questions, as computed by M-Elo, are passed to 
the recommendation engine of the adaptive system. For each 
student, the recommendation engine recommends questions 


?multi-KC in Table 1 indicates the number of items tagged with two 
or more knowledge components (KCs) 
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Figure 2: Overview of the OLM visualising the current knowledge state of a student. 


that are on concepts where a student has the largest knowl- 
edge gap at a difficulty level that reflect the current learn- 
ing ability of the student. This adaptivity at a concept-level 
is because of the availability of the additional parameters 
learned by M-Elo, which is not possible to achieve using the 
standard Elo rating. 


Insights from student feedback. To capture students’ per- 
spectives about the model, a survey was conducted at the 
end of the semester using the following three statements: (1) 
Motivation: the visualisation used by RiPPLE for showing 
my knowledge state increases my motivation to study or fur- 
ther use the system, (2) Rationality: having the ability to 
visually see my current knowledge state, helps me to under- 
stand the rationale behind suggestions made by the system, 
(3) Trust: having the ability to visually see my knowledge 
sate, increases my trust in recommended questions. Re- 
sponses were captured using the Likert scale, where 1 rep- 
resents “strongly disagree” and 5 represents “strong agree”. 
The survey also had an open-ended question asking students 
for feedback on the system. Overall, 55 students who had 
enrolled in the course and used the system voluntarily par- 
ticipated in the survey. Figure 3 represents the results of the 
survey. The majority of students (63.6%) agreed or strongly 


Motivation 305 a ee | 32.70% 
Rationality | 73.60% 38.20% 
Trust | 18.20% 43.60% 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


Strongly Agree Agree — Neutral Disagree i Strongly Disagree 


Figure 3: Student survey results 


agreed that visualisation of their learner model in RiPPLE 
increased their motivation to further use the system. Fur- 
thermore, the majority of the students (61.8%) agreed that 
having the ability to visually see their current knowledge 
state helped them to understand the rationale behind rec- 
ommendations made by the system and that it increased 
their trust in recommendations. In their written comments 
one student mentioned that “I have used a similar platform 
in the past, however, the visualisation of my knowledge state 
in this platform is a great improvement on those.” Inter- 
estingly, two students mentioned that they lost motivation 
in answering easy questions as the potentially large loss of 
rating in answering the question incorrectly outweighed the 
small rating gain received in answering the question cor- 
rectly. Upon closer examination, we noticed that it seemed 


challenging to maintain a balance between the students’ pro- 
ficiency level and the learning items’ difficulty level in our 
pilot. Throughout the semester, the average difficulty level 
of the learning items was falling and the average rating of the 
students’ proficiency was rising. This can be explained by 
the student-centred design of the system that provided stu- 
dents full access to the internet, textbooks and colleagues as 
well as an infinite amount of time for answering a question. 
This may suggest that the zero-sum game principles where 
the net change in ratings after a student has answered a 
question is zero might not be ideal for adaptive learning sys- 
tems. We believe that this may not be an issue for adaptive 
testing, where the exam-like setting of the system might bal- 
ance between the students’ ratings and the learning items’ 
ratings. 


Insights from instructor feedback. To capture the teach- 
ing staff’s perspectives, we held informal discussion sessions 
with members of the teaching staff. They appreciated the 
additional insight on course level and individual level com- 
petencies and gaps that was provided by M-Elo; similar 
benefits have also been reported by [19]. For example, the 
learning model presented in Figure 2 indicates that at a 
class level, students have performed best on “ER-models” 
and “SQL” and have performed worst on “Map-ER-Schema” 
and “Map-Schema-ER” in the course presented in this pilot. 
A shortcoming that the teaching team has noticed was that 
students tended to get discouraged from using the system 
once they had used it for a while as they were not able to 
make significant changes to their knowledge state despite 
answering questions correctly. Upon closer examination, we 
realised that this is due to the use of the uncertainty func- 
tion, as described in section 3 and Formula 4, which reduces 
the sensitivity of the estimations as the number of attempts 
is increased. This functionality seems to be better suited 
for adaptive testing rather than adaptive learning systems 
because of the following reason: Given that a student would 
often take an adaptive test in one sitting with a short time- 
line and receive no feedback on their work, it is common for 
adaptive testing systems to assume no learning has occurred 
during the exam, and as such use of an uncertainty function 
can help with stabilising the ratings; however, a student 
would often interact with an adaptive learning environment 
over a period of time and competencies might improve via 
receiving rich feedback on their learning or decline as a result 
of forgetting. As a consequence, adaptive systems commonly 
expect the knowledge state of the student to change signif- 
icantly as they interact with the system. This means that 
reducing the sensitivity of the estimations over time may 
restrict the model from unwaveringly evolving to accurately 
represent the current knowledge state. 
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5. CONCLUSIONS AND FUTURE DIREC- 
TIONS 


The aim of this paper was to introduce a new multivari- 
ate Elo-based learner model called multi-concept multivari- 
ate Elo rating system (M-Elo) where learning items can be 
tagged with one or more concepts and investigate its ben- 
efits and shortcomings in the context of adaptive learning. 
The results from experiments using a suite of synthetic data 
sets demonstrate that the behaviour of M-Elo is consistent 
with expectations and outperforms the standard Elo-based 
model (Elo) in parameter settings that better reflect real- 
world environments. The results from experiments on mul- 
tiple benchmarking public data sets indicate that M-Elo 
has slightly superior predictive performance compared to 
Elo. Conducting a case study suggested that using M-Elo in 
adaptive learning systems has additional advantages beyond 
using Elo. The first advantage lies in the ability of the model 
in estimating concept-level competencies which can be used 
for guiding adaptivity. The second advantage lies in the abil- 
ity of M-Elo to be opened based on the principles of OLMs. 
Making M-Elo accessible to students increases their moti- 
vation to use the platform and increases their trust in the 
recommendations provided by the platform. It also provides 
additional insight for instructors on individual student-level 
or class-level gaps and competencies that can be used to im- 
prove item and course design. The conducted case study 
gave additional insights into the adverse effects of the zero- 
sum game design of the Elo-based models as well as using 
an uncertainty functions in the context of adaptive learning. 
Future work aims to investigate possible extensions of Elo- 
based models using these considerations and evaluate their 
fit for adaptive learning systems. In addition, quantitative 
comparison of M-Elo and traditional models of learner mod- 
els such as AFM and BKT is required to determine its com- 
petitiveness with traditional models in predicting students’ 
performance. 
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